Systems and methods for emulating far-range lighting for an operational scene illuminated by close-range light

ABSTRACT

A lighting emulation system accesses an image that is captured by an imaging device and that depicts an operational scene illuminated by close-range light. The lighting emulation system accesses a depth map of the operational scene that includes depth data corresponding to each pixel in the image. Based on the depth map, the lighting emulation system determines a far-range lighting coefficient for each pixel in the image. Specifically, the far-range lighting coefficient for each respective pixel is determined based on the corresponding depth data included in the depth map for that respective pixel. Based on the image and the far-range lighting coefficient for each pixel in the image, the lighting emulation system generates a processed image depicting the operational scene as illuminated by far-range lighting, and provides the processed image for presentation on a display screen.

RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication No. 62/769,481, filed on Nov. 19, 2018, and entitled“SYSTEMS AND METHODS FOR EMULATING FAR-RANGE LIGHTING FOR AN OPERATIONALSCENE ILLUMINATED BY CLOSE-RANGE LIGHT,” the contents of which arehereby incorporated by reference in their entirety.

BACKGROUND INFORMATION

During a computer-assisted surgical procedure, such as a minimallyinvasive surgical procedure that uses a computer-assisted surgicalsystem, an imaging device such as an endoscope or other similarinstrument may be used to capture imagery of an operational scene withinan operational area (e.g., an area in which an operation such as asurgical procedure, an imaging operation, or another type of operationis being performed). The computer-assisted surgical system may displaythe captured imagery to medical personnel (e.g., to a surgeon and/orother members of a surgical team) to provide a visualization of theoperational area that may assist the medical personnel in performing anoperation at the operational area. However, there remains room toimprove imagery of operational areas and the technologies used toprovide such imagery.

SUMMARY

Systems and methods for emulating far-range lighting for an operationalscene illuminated by close-range light are described herein. Forinstance, one embodiment is implemented as a system comprising aprocessor and a memory communicatively coupled with the processor andstoring instructions that are executable by the processor. Theinstructions, when executed by the processor, may direct the processorto access an image that is captured by an imaging device and thatdepicts an operational scene illuminated by close-range light, as wellas to access a depth map of the operational scene. The depth map mayinclude depth data corresponding to each pixel in the image, and theinstructions, when executed, may further direct the processor todetermine a far-range lighting coefficient for each pixel in the imagebased on the depth map. For example, the far-range lighting coefficientfor each respective pixel may be determined based on the correspondingdepth data included in the depth map for that respective pixel. Based onthe image and the far-range lighting coefficient for each pixel in theimage, the processor may further be directed to generate a processedimage depicting the operational scene as being illuminated by far-rangelighting, and to provide the processed image for presentation on adisplay screen.

Another exemplary embodiment is implemented as a system comprisingvarious elements. For instance, the system may comprise an imagingdevice configured to capture an image depicting an operational scene asilluminated by close-range light, a physical light source associatedwith the imaging device and configured to generate the close-range lightto illuminate the operational scene, a display screen configured topresent images captured by the imaging device, a processorcommunicatively coupled to the imaging device, and a memorycommunicatively coupled to the processor and storing instructionsexecutable by the processor. In this embodiment, the instructions, whenexecuted, may be configured to direct the processor to direct theimaging device to capture the image depicting the operational scene asilluminated by the close-range light, as well as to generate, based onthe image depicting the operational scene, a depth map of theoperational scene. The depth map may include depth data corresponding toeach pixel in the image, and the instructions, when executed, mayfurther direct the processor to determine a far-range lightingcoefficient for each pixel in the image based on the depth map. Forexample, the far-range lighting coefficient for each respective pixelmay be determined based on the corresponding depth data included in thedepth map for that respective pixel. Based on the image and thefar-range lighting coefficient for each pixel in the image, theprocessor may further be directed to generate a processed imagedepicting the operational scene as being illuminated by far-rangelighting, and to provide the processed image for presentation on thedisplay screen.

Yet another exemplary embodiment is implemented as a method performed bya lighting emulation system. For example, the method includes accessingan image that is captured by an imaging device and that depicts anoperational scene illuminated by close-range light, as well as accessinga depth map of the operational scene, the depth map including depth datacorresponding to each pixel in the image. The method further includesdetermining, based on the depth map, a far-range lighting coefficientfor each pixel in the image, wherein the determining of the far-rangelighting coefficient for each respective pixel is performed based on thecorresponding depth data included in the depth map for that respectivepixel. The method also includes generating, based on the image and thefar-range lighting coefficient for each pixel in the image, a processedimage depicting the operational scene as being illuminated by far-rangelighting, as well as providing the processed image for presentation on adisplay screen.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various embodiments and are a partof the specification. The illustrated embodiments are merely examplesand do not limit the scope of the disclosure. Throughout the drawings,identical or similar reference numbers designate identical or similarelements.

FIG. 1 illustrates an exemplary computer-assisted surgical systemaccording to principles described herein.

FIG. 2 illustrates an exemplary imaging device located at an operationalarea associated with a patient according to principles described herein.

FIGS. 3A and 3B illustrate exemplary aspects of an inverse square lawthat describes the brightness of light as a function of distance from alight source that originates the light according to principles describedherein.

FIG. 4 illustrates an exemplary lighting emulation system for emulatingfar-range lighting for an operational scene illuminated by close-rangelight according to principles described herein.

FIG. 5 illustrates an exemplary configuration in which the lightingemulation system of FIG. 4 may operate to emulate far-range lighting foran operational scene illuminated by close-range light according toprinciples described herein.

FIGS. 6 and 7 illustrate exemplary brightness values of certain pixelsof respective exemplary images that depict, in accordance with differentrespective exposure algorithms, an operational scene illuminated byclose-range light according to principles described herein.

FIG. 8 illustrates exemplary depth data stored in an exemplary depth mapaccording to principles described herein.

FIG. 9 illustrates an exemplary dataflow for operations performed by anexemplary implementation of the lighting emulation system of FIG. 4according to principles described herein,

FIG. 10 illustrates exemplary brightness values of certain pixels of anexemplary processed image after a raw image is processed in accordancewith the dataflow of FIG. 9 according to principles described herein.

FIG. 11 illustrates an exemplary method for emulating far-range lightingfor an operational scene illuminated by close-range light according toprinciples described herein.

FIG. 12 illustrates an exemplary computing device according toprinciples described herein.

DETAILED DESCRIPTION

Systems and methods for emulating far-range lighting for an operationalscene illuminated by close-range light are described herein. Exemplarylighting emulation systems and methods described herein may beimplemented by a computer-assisted surgical system and may improvebrightness uniformity in images captured by imaging devices that captureimages from a close range using close-range light. For example, as willbe described in more detail below, an imaging device such as anendoscope or similar instrument may capture an image of an operationalscene (e.g., a surgical scene including internal anatomy of a patient)that is illuminated by light originating from the imaging device itselfor another light source near the operational scene (e.g., within a fewcentimeters of the operational scene). Due to the inverse square law oflight (described below), the brightness of such images may tend to benon-uniform in brightness, leading to various issues when the images arepresented. Accordingly, systems and methods described herein improve thebrightness uniformity of such images based on depth data associated withthe operational scene, and in such a way that the resulting imagesappear natural, consistent, and/or attractive. In some implementations,for instance, images of an operational scene illuminated by close-rangelight may be processed so as to give an appearance that the operationalscene was instead illuminated by far-range light originating from avirtual light source relatively far away from the operational scene(e.g., at a location emulating a location of a light source that wouldbe used in an open surgery).

To this end, in certain examples, an exemplary lighting emulation systemmay include or be implemented by a processor and a memory that iscommunicatively coupled with the processor and that stores instructionsthat are executable by the processor to direct the processor to performvarious operations associated with emulating the far-range lighting forthe operational scene illuminated by the close-range light. Suchoperations may include, for instance, any of the operations that willnow be described.

The lighting emulation system may access an image depicting anoperational scene illuminated by close-range light. The image may becaptured by an imaging device (e.g., by way of an image sensor includedwithin the imaging device). The imaging device may be located at anoperational area that includes the operational scene. In an exemplaryscenario involving a computer assisted surgery such as a minimallyinvasive surgery, the operational area may be an internal space of thepatient, the operational scene may include particular anatomy within theoperational area that is being imaged and/or operated on, the imagingdevice may be an endoscope or similar instrument (e.g., a laparoscope, ahyperspectral imaging device, etc.), and the image sensor may be asensor associated with the imaging device and configured to captureimages depicting the operational scene.

The lighting emulation system may further access a depth map of theoperational scene. The depth map may include depth data corresponding toeach pixel in the image. As will be described in more detail below, thedepth map may be accessed in various ways such as by being received fromanother device or system, or by being generated by the lightingemulation system based on an image or images captured or accessed by thelighting emulation system itself. Based on the depth map, the lightingemulation system may determine a far-range lighting coefficient for eachpixel in the image. For example, the far-range lighting coefficient foreach respective pixel may be determined based on the corresponding depthdata included in the depth map for that respective pixel. Based on theimage and the far-range lighting coefficient for each pixel in theimage, the lighting emulation system may generate a processed imagedepicting the operational scene as being illuminated by far-rangelighting. The lighting emulation system may provide this processed imagefor presentation on a display screen.

Various benefits may be provided by the lighting emulation systems andmethods described herein. For example, as mentioned above, brightnessuniformity may be improved for captured images with highly non-uniformbrightness or a wide dynamic range of brightness. By increasing thebrightness uniformity and/or reducing the dynamic range of brightness inthe ways described herein, lighting emulation systems and methods mayprovide images that, when presented on standard display screens (e.g.,LCD monitors, etc.), look more natural and attractive, show moreinformation, show information that is more readily understandable, andso forth compared to images not processed in accordance with systems andmethods described herein. Images captured by endoscopic imaging devicessuch as those described herein may be particularly prone to having lowbrightness uniformity and wide dynamic range, thus commonly leading toand/or exacerbating various issues described herein. As will bedescribed in more detail below, this is partly due to an inverse squarelaw that defines the brightness of light as a function of distance froma light source. As such, many images captured using endoscopic imagingdevices are likely to benefit significantly from the methods describedherein of increasing brightness uniformity and reducing dynamic range.

Another advantage that may be provided by the systems and methodsdescribed herein relates to the manner in which brightness uniformity isincreased. Rather than decreasing the brightness of initially brightpixels and/or increasing the brightness of initially dim pixels basedonly on raw brightness levels of the pixels, as performed byconventional systems for increasing brightness uniformity, methods andsystems described herein adjust the brightness of each pixel of an imagein accordance with a far-range lighting coefficient that is determinedbased on depth data corresponding to each pixel. This provides a finalprocessed image that looks more natural (e.g., more genuine andrealistic, less artificial or processed, etc.) than images that arealtered without taking depth into account. By utilizing depth data asdescribed herein, systems and methods described herein may emulate,within an image of an operational scene, far-range lighting such aswould be present in an open surgery or other such scenario in whichlight originates from a location far enough away from the operationalscene that the inverse square law of light does not have a significantor noticeable effect.

Along with an increase in uniformity of brightness, systems and methodsof adjusting brightness that account for depth, such as the lightingemulation systems and methods described herein, may also beneficiallyprovide an increase in brightness consistency. For example, rather thanan entire image becoming relatively bright and then going relatively dimas the imaging device moves over dimmer and brighter sections of anoperational scene, as may occur with using conventional autoexposurealgorithms, the brightness of processed images provided by the systemsand methods described herein may remain consistently as bright as a usermight want them to be. For example, the brightness may consistently beas bright as simulated far-range light associated with open surgerylighting at a particular distance from the operational scene, or asbright as a user may select (e.g., in accordance with the user'spreferences).

By emulating far-range lighting for an operational scene illuminated byclose-range light to provide any of these or other benefits of improvingbrightness uniformity, systems and methods described herein mayfacilitate surgeons and other medical personnel in performing their workaccurately, efficiently, and effectively. In this way, the medicalpersonnel may enjoy an improved experience with fewer distractionsduring operations they perform. In turn, these improved experiences ofthe people performing the operations may lead to more effective andefficient operations and improved patient outcomes.

Various embodiments will now be described in more detail with referenceto the figures. The systems and methods described herein may provide oneor more of the benefits mentioned above and/or various additional and/oralternative benefits that will be made apparent herein.

Lighting emulation systems and methods described herein may operate aspart of or in conjunction with a computer-assisted surgical system. Assuch, in order to promote an understanding of lighting emulation systemsand methods described herein, an exemplary computer-assisted surgicalsystem will now be described. The described exemplary computer-assistedsurgical system is illustrative and not limiting. Lighting emulationsystems and methods described herein may operate as part of or inconjunction with the computer-assisted surgical system described hereinand/or with other suitable computer-assisted surgical systems.

FIG. 1 illustrates an exemplary computer-assisted surgical system 100(“surgical system 100”). As shown, surgical system 100 may include amanipulating system 102, a user control system 104, and an auxiliarysystem 106 communicatively coupled one to another. Surgical system 100may be utilized by a surgical team to perform a computer-assistedsurgical procedure on a patient 108. As shown, the surgical team mayinclude a surgeon 110-1, an assistant 110-2, a nurse 110-3, and ananesthesiologist 110-4, all of whom may be collectively referred to as“surgical team members 110.” Additional or alternative surgical teammembers may be present during a surgical session as may serve aparticular implementation.

While FIG. 1 illustrates an ongoing minimally invasive surgicalprocedure, it will be understood that surgical system 100 may similarlybe used to perform open surgical procedures or other types of surgicalprocedures that may similarly benefit from the accuracy and convenienceof surgical system 100. Additionally, it will be understood that thesurgical session throughout which surgical system 100 may be employedmay not only include an operative phase of a surgical procedure, as isillustrated in FIG. 1, but may also include preoperative, postoperative,and/or other such phases of the surgical procedure. A surgical proceduremay include any procedure or operation in which manual and/orinstrumental techniques are used on a patient to investigate or treat aphysical condition of the patient.

As shown in FIG. 1, manipulating system 102 may include a plurality ofmanipulator arms 112 (e.g., manipulator arms 112-1 through 112-4) towhich a plurality of surgical instruments may be coupled. Each surgicalinstrument may be implemented by any suitable surgical tool (e.g., atool having tissue-interaction functions), medical tool, monitoringinstrument (e.g., an imaging device such as an endoscope), sensinginstrument (e.g., a force-sensing surgical instrument), diagnosticinstrument, or the like that may be used for a computer-assistedsurgical procedure on patient 108 (e.g., by being at least partiallyinserted into patient 108 and manipulated to perform a computer-assistedsurgical procedure on patient 108). While manipulating system 102 isdepicted and described herein as including four manipulator arms 112, itwill be recognized that manipulating system 102 may include only asingle manipulator arm 112 or any other number of manipulator arms asmay serve a particular implementation.

Manipulator arms 112 and/or surgical instruments attached to manipulatorarms 112 may include one or more displacement transducers, orientationalsensors, and/or positional sensors used to generate raw (i.e.,uncorrected) kinematics information. One or more components of surgicalsystem 100 may be configured to use the kinematics information to track(e.g., determine positions of) and/or control the surgical instruments.

Surgical instruments attached to manipulator arms 112 may each bepositioned at an operational area associated with a patient. As usedherein, an “operational area” or a “surgical area” associated with apatient may, in certain examples, be entirely disposed within thepatient and may include an area within the patient near where anoperation (e.g., a surgical procedure) is planned to be performed, isbeing performed, or has been performed. For example, for a minimallyinvasive surgical procedure being performed on tissue internal to apatient, the operational area may include the tissue, anatomy underlyingthe tissue, as well as space around the tissue where, for example,surgical instruments being used to perform the operation are located. Inother examples, an operational area may be at least partially disposedexternal to the patient. For instance, surgical system 100 may be usedto perform an open surgical procedure such that part of the operationalarea (e.g., tissue being operated on) is internal to the patient whileanother part of the operational area (e.g., a space around the tissuewhere one or more surgical instruments may be disposed) is external tothe patient. A surgical instrument may be referred to as being locatedat or within an operational area when at least a portion of the surgicalinstrument (e.g., a distal end of the surgical instrument) is locatedwithin the operational area.

User control system 104 may be configured to facilitate control bysurgeon 110-1 of manipulator arms 112 and surgical instruments attachedto manipulator arms 112. For example, surgeon 110-1 may interact withuser control system 104 to remotely move or manipulate manipulator arms112 and the surgical instruments. To this end, user control system 104may provide surgeon 110-1 with imagery (e.g., high-definition 3Dimagery) of an operational area associated with patient 108 as capturedby an imaging device. In certain examples, user control system 104 mayinclude a stereo viewer having two displays where stereoscopic images ofan operational scene included within an operational area associated withpatient 108 and generated by a stereoscopic imaging device may be viewedby surgeon 110-1. Surgeon 110-1 may utilize the imagery to perform oneor more procedures with one or more surgical instruments attached tomanipulator arms 112.

To facilitate control of surgical instruments, user control system 104may include a set of master controls. These master controls may bemanipulated by surgeon 110-1 to control movement of surgical instruments(e.g., by utilizing robotic and/or teleoperation technology). The mastercontrols may be configured to detect a wide variety of hand, wrist, andfinger movements by surgeon 110-1. In this manner, surgeon 110-1 mayintuitively perform a procedure using one or more surgical instruments.

Auxiliary system 106 may include one or more computing devicesconfigured to perform primary processing operations of surgical system100. In such configurations, the one or more computing devices includedin auxiliary system 106 may control and/or coordinate operationsperformed by various other components (e.g., manipulating system 102 anduser control system 104) of surgical system 100. For example, acomputing device included in user control system 104 may transmitinstructions to manipulating system 102 by way of the one or morecomputing devices included in auxiliary system 106. As another example,auxiliary system 106 may receive, from manipulating system 102, imagedata representative of imagery captured by an imaging device attached toone of manipulator arms 112, and may process the image data in any ofthe ways described herein.

In some examples, auxiliary system 106 may be configured to presentvisual content to surgical team members 110 who may not have access tothe images provided to surgeon 110-1 at user control system 104. To thisend, auxiliary system 106 may include a display monitor 114 configuredto display one or more user interfaces, such as images (e.g., 2D images)of the operational area, information associated with patient 108 and/orthe surgical procedure, and/or any other visual content as may serve aparticular implementation. For example, display monitor 114 may displayimages of an operational scene included within the operational areatogether with additional content (e.g., graphical content, contextualinformation, etc.) concurrently displayed with the images. In someembodiments, display monitor 114 is implemented by a touchscreen displaywith which surgical team members 110 may interact (e.g., by way of touchgestures) to provide user input to surgical system 100.

Manipulating system 102, user control system 104, and auxiliary system106 may be communicatively coupled one to another in any suitablemanner. For example, as shown in FIG. 1, manipulating system 102, usercontrol system 104, and auxiliary system 106 may be communicativelycoupled by way of control lines 116, which may represent any wired orwireless communication link as may serve a particular implementation. Tothis end, manipulating system 102, user control system 104, andauxiliary system 106 may each include one or more wired or wirelesscommunication interfaces, such as one or more local area networkinterfaces, Wi-Fi network interfaces, cellular interfaces, etc.

FIG. 2 illustrates an exemplary imaging device 200 located at anoperational area associated with a patient. Specifically, as shown,imaging device 200 may be implemented as a stereoscopic endoscope.Imaging device 200 may be manually controlled (e.g., by a surgeonperforming an operation on a patient). Alternatively, imaging device 200may be coupled to a manipulator arm (e.g., one of manipulator arms 112)of a computer-assisted surgical system (e.g., surgical system 100), andcontrolled using robotic and/or teleoperation technology. Imaging device200 is representative of many different types and/or implementations ofendoscopes and other similar imaging tools that may be used with systemsand methods described herein.

As shown, imaging device 200 includes a shaft 202 and a camera head 204coupled to a proximal end of shaft 202. Camera head 204 is configured tobe located external to the patient. Shaft 202 has a distal end that isconfigured to be positioned at (e.g., inserted into) an operational areaassociated with a patient. In various implementations, shaft 202 isrigid (as shown in FIG. 2). Alternatively, shaft 202 may be jointedand/or flexible.

As shown in the stereoscopic implementation of imaging device 200 inFIG. 2, camera head 204 houses a right-side camera control unit 206-R, aleft-side camera control unit 206-L, and an illuminator 208. In somealternative examples, camera control units 206 and illuminator 208 arenot included in camera head 204 and are instead located in a controllerdevice communicatively coupled to imaging device 200. The controllerdevice may be implemented by auxiliary system 106, for example.

Shaft 202 houses a right-side image sensor 210-R optically coupled to aright-side optic 212-R, a left-side image sensor 210-L optically coupledto a left-side optic 212-L, and an illumination channel 214.Collectively, the right-side components (i.e., camera control unit206-R, image sensor 210-R, and optic 212-R) implement a camera thatcaptures images 216-R of an operational scene (e.g., including anatomylocated within the operational area) from a right-side perspective.Likewise, the left-side components (i.e., camera control unit 206-L,image sensor 210-L, and optic 212-L) collectively implement a camerathat captures images 216-L of the operational scene from a left-sideperspective.

To capture images 216, illuminator 208 generates light, which is carriedby one or more optical fibers in illumination channel 214 and outputinto the operational area at a distal end of shaft 202 so as toilluminate the operational scene. Optics 212, which may each beimplemented by a lens or other suitable component, capture the lightafter the light reflects from patient anatomy and/or other objectswithin the operational scene. In some examples, light used to captureimages of the operational scene may originate from another physicallight source (e.g., from a different instrument such as a differentimaging device, a dedicated lighting tool, or the like). However,because the operational area is internal to the patient (i.e., beneaththe skin of the patient, as shown), any physical light sourceilluminating the operational scene may be located relatively close to(e.g., within a few centimeters of) the operational scene. As such, theoperational scene in such internal operational areas may be illuminatedby close-range light. As will be described in more detail below, thisclose-range lighting of the operational scene may result in images 216-Rand 216-L that have relatively high dynamic ranges and relatively lowbrightness uniformity, thereby making the images difficult to see anddisplay (e.g., because the images have various areas that are either toodark or too bright for viewers to easily make out details beingdepicted). Accordingly, images capturing the operational scene asilluminated by the close-range light of illuminator 208 and illuminationchannel 214 may be processed by systems for emulating far-range lightingdescribed herein.

The light captured by optics 212 is sensed by image sensors 210. Imagesensors 210 may be implemented as any suitable image sensors such ascharge coupled device (“CCD”) image sensors, complementary metal-oxidesemiconductor (“CMOS”) image sensors, or the like. Image sensors 210-Rand 210-L convert the sensed light into signals (e.g., video data)representative of images, and transmit the signals to camera controlunits 206 by way of conduits 218-R and 218-L, respectively. Conduits 218may be any suitable communication link configured to handle high-speedtransmission of data.

Camera control units 206 process the signals received from image sensors210 and generate, based on the signals, data representative of images216. Camera control units 206 then transmit the data to an externaldevice (e.g., a computing device that processes the images and/ordisplays the images and/or video formed by the images on a displayscreen). As shown, camera control units 206 are synchronously coupled toone another by way of a communicative link 220 so that images 216 aresynchronized.

Additional or alternative components may be included in imaging device200. For example, one or more other optics not explicitly shown in FIG.2 may be included in shaft 202 for focusing, diffusing, transmitting, orotherwise treating light generated and/or sensed by imaging device 200.For instance, in some alternative examples, image sensors 210 may bepositioned closer to the proximal end of shaft 202 or inside camera head204, a configuration commonly referred to as a rod lens endoscope. Inthese examples, optics may carry light captured at the distal end ofshaft 202 along shaft 202 to reach image sensors 210 at their respectivelocations at a more proximal part of shaft 202, at camera head 204, orat any other location where the image sensors may be located in aparticular implementation.

Imaging device 200 may provide data representing visible light capturedat an operational scene of an operational area. For example, imagingdevice 200 may provide data representative of visible light images ofthe operational scene sensed by imaging device 200. Visible light imagesmay include or be implemented as images using any suitable color and/orgrayscale palette to represent a visible light-based view of theoperational scene.

Imaging device 200 may also determine and provide data representingdepth data of the operational scene, or data that may be processed toderive such depth data. For example, imaging device 200 may capture andprovide images of the operational scene that represent depth sensed byimaging device 200. Alternatively, imaging device 200 may capture imagesof the surgical area that may be processed to derive depth data of thesurgical area. For example, images 216-R and 216-L may be stereoscopicimages of the operational scene, which images may be processed todetermine depth information for the operational scene. The depthinformation may be represented as a depth map (e.g., a representation ofthe operational scene obtained using a Z-buffer that indicates distancefrom imaging device 200 to each pixel in the representation), which maybe configured to indicate depths of objects in any suitable way, such asby using different greyscale values to represent different depth values.

Images captured by imaging device 200 and/or derived from imagescaptured by imaging device 200 (e.g., visible light images, depth maps,etc.) may be referred to as “imaging device imagery.” Exemplary lightingemulation systems and methods described herein may be configured toutilize imaging device imagery to provide visualizations of anatomicalstructures, such as described herein.

Imaging device 200 shown in FIG. 2 is illustrative of one imaging devicethat may be used to obtain imaging device imagery. Any other suitableimaging device or combination of devices from which visible light dataand/or depth data of an operational area may be obtained or derivedduring an operation may be used in other examples.

As mentioned above, a captured image of an operational scene illuminatedby close-range light (such as light originating from illuminator 208 andillumination channel 214 in FIG. 2) may tend to have a low uniformity ofbrightness across the various pixels in the image due, at least in part,to an inverse square law that describes how brightness of lightdecreases with respect to distance from a light source. Mathematically,this inverse square law is applicable to point sources of light.However, it will be understood that the inverse square law provides ahelpful and accurate approximation of brightness associated with lightsources that may approximate or be modeled as point sources (e.g., suchas illuminator 208 and illumination channel 214 of imaging device 200, areflection of light from a particular point in an operational scene,etc.), even if such sources are not true point sources. As such, theinverse square law described herein will be understood to be applicableto endoscopic and other light sources described herein, even if,technically speaking, the law describes models that only approximatelydescribe these light sources.

The inverse square law of light dictates that brightness of lightdecreases with distance from the light source in a non-linear manner.Specifically, the brightness decreases in accordance with an inversesquare curve described by an equation of the form of Equation 1, below.

$\begin{matrix}{{Brightness} = \frac{1}{({Distance})^{2}}} & ( {{Equation}\mspace{14mu} 1} )\end{matrix}$

Certain consequences arise from the brightness of light dropping offwith the square of distance from a light source as described inEquation 1. For instance, objects located at a relatively far distancefrom a light source may be illuminated with great brightness uniformitybecause variations in distance of different points on the surfaces ofthe objects from the light source may be negligible. If one point on theobject surface is 99 distance units (e.g., 99 cm) from the light source,for example, and another point on the object surface is 100 distanceunits (e.g., 100 cm) from the light source, Equation 1 shows that thedifference in brightness with which these points will be illuminated maybe very small (e.g., 1/(99²)/1/(100²)=1.02030, representing onlyapproximately a 2% difference in brightness illuminating these points).Conversely, however, objects located relatively close to the lightsource may be illuminated with dramatically decreased brightnessuniformity because variations in distance of the different points fromthe light source may be significant. For example, taking the same objectwith two surface points one distance unit apart and moving the objecttoward the light source so that one of the points is one distance unit(e.g., 1 cm) from the light source and the other point is two distanceunits (e.g., 2 cm) from the light source, Equation 1 shows that thedifference in brightness with which the points are illuminated may bemuch more significant (e.g., 1/(1²)/1/(2²)=4.00000, representing adifference in brightness where one point is four times brighter than theother point).

Due to this inverse square law, one consequence for capturing imagesilluminated by close-range light is that a much greater dynamic rangemay be required to capture, present, and/or view the images than if theimages were illuminated by far-range light. This is because, when asurface is relatively close to a light source, relatively small changesin distance of surface points may correspond to relatively large changesin brightness with which the surface points are illuminated. Tofacilitate presenting captured images of an operational scene, it maythus be desirable for the operational scene depicted in the images to beilluminated by relatively far-range light, such that small changes indistance of surface points may correspond to small (e.g., negligible)changes in brightness with which the surface points are illuminated.However, because it may not be possible, due to the location of anoperational scene within an internal operational area, to physicallyilluminate the operational scene with far-range light, systems andmethods described herein may perform processing operations to emulatefar-range lighting for operational scenes illuminated by close-rangelight.

FIGS. 3A and 3B illustrate exemplary aspects of the inverse square lawdescribed above in relation to Equation 1 and how the inverse square lawrelates to the emulation, by systems and methods described herein, offar-range lighting for operational scenes illuminated by close-rangelight.

Specifically, FIG. 3A shows an exemplary virtual light source 302 thatmay be modeled as a point source of light and that is originating asector of light 304 from a virtual position P0 (e.g., a virtual positionin space in relation to a position of an operational scene). As shown,light 304 naturally spreads out as it travels from the P0 positionoutward to other labeled positions (e.g., positions P1, P2, P3, and P4)and beyond. Each of the labeled positions P1 through P4 are shown to bea certain distance 308 apart, and respective portions 306 of anoperational scene (i.e., portions 306-1 through 306-4) are shown torepresent different-sized portions of the operational scene that arecovered by the sector of light 304 if the operational scene were to belocated at respective distances of respective positions P1 through P4.For example, at one distance 308 from position P0 (i.e., at positionP1), a portion 306-1 having a size of one unit area (i.e., one square)would be large enough to capture the entirety of the sector of light304. However, at two distances 308 from position P0 (i.e., at positionP2), light 304 is shown to spread over a portion 306-2 that has a sizeof not two unit areas, but 2² unit areas (i.e., 4 squares). Accordingly,as illustrated by the shaded squares, an area the same size as theentire portion 306-1, when positioned at position P2, receives onlyone-fourth of the light 304 (i.e., is illuminated with one-fourth thebrightness). Similarly, a shaded square in portion 306-3 is shown to beilluminated with only one-ninth the brightness of the same sized area ifthe operational scene is positioned three distances 308 from position P0(i.e., at position P3), a shaded square in portion 306-4 is shown to beilluminated with only one-sixteenth the brightness of the same sizedarea if the operational scene is positioned four distances 308 fromposition P0 (i.e., at position P4), and so forth.

As notated in FIG. 3A, when an operational scene (or portion thereofsuch as a portion 306) is at a relatively great distance from a lightsource such as virtual light source 302, the lighting of the operationalscene by the light source may be considered to be far-range lighting.Conversely, when an operational scene (or portion thereof) is at arelatively small distance from the light source, the lighting of theoperational scene by the light source may be considered to beclose-range lighting. Examples of close-range lighting and far-rangelighting, as well as how a scene illuminated by close-range light may bemade to emulate being illuminated by far-range light, will be describedin more detail below.

FIG. 3B depicts a graph illustrating the inverse square law ofbrightness. Specifically, the graph includes a curve 310 that is plottedwith distance of the operational scene from virtual light source 302along the x-axis and brightness per unit area along the y-axis. Asshown, at position P1 (i.e., at one distance 308 from position P0 wherelight 304 originates), the brightness is one brightness unit, signifyingthat one unit area receives 100% of the sector of light 304 and is thusat what may be defined for this example as 100% brightness. At positionP2 (i.e., at two distances 308 from position P0), the brightness perunit area (e.g., the brightness of one square such as the shaded square)is one-fourth of a brightness unit, signifying that each unit areareceives only 25% of the sector of light 304 and is thus at 25%brightness. Similarly, at position P3 (i.e., at three distances 308 fromposition P0), the brightness per unit area is one-ninth of a brightnessunit, signifying that each unit area receives only about 11% of light304 and is thus at about 11% brightness; at position P4 (i.e., at fourdistances 308 from position P0), the brightness per unit area isone-sixteenth of a brightness unit, signifying that each unit areareceives only about 6% of light 304 and is thus at about 6% brightness;and so forth.

As shown in FIG. 3B, while curve 310 has a steep slope near the y-axis(i.e., near position P0 where light 304 originates), the slope is shownto level off significantly at greater distances. The slope of curve 310may thus be related to the brightness uniformity and dynamic range ofbrightness with which the operational scene may be illuminated by a realor virtual light source (e.g., light source 302) when the light sourceis positioned at a particular distance from the operational scene.Specifically, as the slope of curve 310 gets flatter and flatter atgreater distances, the dynamic range of captured images may become lessand less and the brightness uniformity greater and greater. This isbecause the brightness difference between different points on a surfaceof an object (e.g., different points having small variations indistance) becomes more and more negligible as curve 310 gets flatter atgreater distances. As used herein, a dynamic range of an image may beconceptually described and/or actually calculated as a ratio of abrightness of a brightest pixel in the image to a brightness of adimmest pixel in the image. In contrast, a brightness uniformity may beconceptually described and/or actually calculated as the inverse of thedynamic range. In some examples (including certain examples describedbelow), a few outlying pixels in an image (e.g., those representing adark crevice, a bright glare, etc.) may be too far to extremes ofbrightness or dimness to be helpful in expressing a meaningful dynamicrange or brightness uniformity value. As such, these pixels may not beaccounted for in determining which pixels are the brightest and/or thedimmest in the image, or otherwise in determining or describing adynamic range or brightness uniformity of an image.

As mentioned above, because operational scenes within internaloperational areas such as the operational area illustrated in FIG. 2 maycommonly (or by necessity) be illuminated by close-range light forimages of the operational scenes to be captured, lighting emulationsystems described herein may be configured to process the images toemulate far-range lighting so as to provide an image with a lowerdynamic range and higher brightness uniformity. This may result in moreattractive and informative images (e.g., images that are brighter buthave fewer saturated pixels) and other advantages described above.

To illustrate one such lighting emulation system, FIG. 4 shows anexemplary lighting emulation system 400 (“system 400”) for emulatingfar-range lighting for an operational scene illuminated by close-rangelight. As shown, system 400 may include, without limitation, a dataaccess facility 402, a data analysis facility 404, a data generationfacility 406, and a storage facility 408 selectively and communicativelycoupled to one another. It will be recognized that although facilities402 through 408 are shown to be separate facilities in FIG. 4,facilities 402 through 408 may be combined into fewer facilities, suchas into a single facility, or divided into more facilities as may servea particular implementation.

Each of facilities 402 through 408 may include or be implemented by oneor more physical computing devices such as hardware and/or softwarecomponents (e.g., processors, memories, communication interfaces,instructions stored in memory for execution by the processors, etc.).The facilities may be implemented using separate computing componentsunique to each facility, or may be implemented using shared computingcomponents. For instance, in certain examples, each of facilities 402through 408 may be distributed between multiple devices as may serve aparticular implementation. Additionally, one or more of facilities 402through 408 may be omitted from system 400 in certain implementations,while additional facilities may be included within system 400 in thesame or other implementations.

In some examples, facilities 402 through 408 may be configured tooperate in real time so as to access and process image data and/or depthdata as the data is captured or otherwise generated. As such, system 400may provide a processed image for presentation on a display screen liveand in real time such that, for example, surgical team members may beconstantly apprised of what is happening in the internal operationalarea as an operation (e.g., a surgical procedure) is performed.Operations of facilities 402 through 408 may be performed in real timewhen they are performed immediately and without undue delay, even ifthere is some amount of processing delay. Each of facilities 402 through408 will now be described in more detail.

Data access facility 402 may be configured to perform various operationsassociated with requesting, communicating, receiving, or otherwiseaccessing input data for processing by system 400. For example, dataaccess facility 402 may be configured to access an image depicting anoperational scene illuminated by close-range light. The image may becaptured, for example, by an image sensor (e.g., one of image sensors210) included within an imaging device (e.g., imaging device 200)located at an operational area that includes the operational scene. Aswill be described in more detail below, the captured image may, in someexamples, be included within a sequence of images, such as by comprisinga single frame in a video sequence or the like. In certain examples,system 400 may be separate from an imaging device such as imaging device200, and may thus access the image by receiving the image from theimaging device. In other examples, system 400 may include or implementan imaging device such as imaging device 200, and may thus access theimage by using the integrated imaging device to capture the image.

Along with accessing image data depicting the operational scene, dataaccess facility 402 may further access other types of data (e.g., othertypes of imaging device imagery) such as depth data included in a depthmap of the operational scene. For example, along with each capturedimage, data access facility 402 may access a depth map that includesdepth data corresponding to each pixel in the accessed image. Depth datamay be accessed in any manner as may serve a particular implementation,including in any of the ways described above or that will be describedbelow in more detail.

Data analysis facility 404 may be configured to perform variousoperations associated with analyzing, processing, calculating,computing, or otherwise manipulating data (e.g., data accessed by dataaccess facility 402) to facilitate the emulation of far-range lightingfor an operational scene illuminated by close-range light. For example,data analysis facility 404 may be configured to determine, based on thedepth map accessed by data access facility 402, a far-range lightingcoefficient for each pixel in the image accessed by data access facility402. Specifically, data analysis facility 404 may determine differentfar-range lighting coefficients for all the pixels in the image, eachcoefficient determined based on depth data indicative of a distance of asurface point from a source of close-range light. For instance, eachrespective pixel in the image may represent a different surface pointincluded in the operational scene, and the far-range lightingcoefficient for each respective pixel may be determined based on thecorresponding depth data included in the depth map for that respectivepixel.

Data generation facility 406 may be configured to perform variousoperations associated with generating and providing processed data basedon the data accessed by data access facility 402 and processed by dataanalysis facility 404. For example, data generation facility 406 may beconfigured to generate a processed image depicting the operational sceneas being illuminated by far-range lighting, and to provide the processedimage for presentation on a display screen (e.g., display monitor 114 ofauxiliary system 106 in surgical system 100, a display screen includedwithin user control system 104 in surgical system 100, etc.). Theprocessed image may be generated and provided, for instance, based onthe image accessed by data access facility 402 and the far-rangelighting coefficients determined by data analysis facility 404 for thepixels included in the image. As will be described in more detail below,by emulating far-range lighting for the operational scene in this way(rather than, for example, merely performing conventional tone mappingoperations to reduce the dynamic range based only on the brightness ofthe captured pixels), data generation facility 406 may generate andprovide a processed image that may be preferred (e.g., over imagesprocessed using conventional dynamic range compression algorithms) bythe surgical team members to whom the processed image is presented.Detailed methods for performing the operations described in relation tofacilities 402 through 406 will be described in more detail below.

Storage facility 408 may maintain any suitable data received, generated,managed, analyzed, processed, used, and/or transmitted by facilities 402through 406 in a particular implementation. For example, storagefacility 408 may store or temporarily buffer raw or processed imagedata, depth data, far-range lighting coefficient data, or other datareceived, generated, managed, maintained, used, and/or transmitted byfacilities 402 through 406. Additionally, storage facility 408 mayinclude program instructions and/or other such data used by facilities402 through 406 to perform any of the operations described herein.

FIG. 5 illustrates an exemplary configuration 500 in which system 400may operate to emulate far-range lighting for an operational sceneilluminated by close-range light. Specifically, as shown, configuration500 includes an imaging device 502 located in an operational area andcapturing images of anatomy 504 associated with an operational scenewithin the operational area. The operational area, as well as theoperational scene being imaged within the operational area, may besimilar to the operational area and the operational scene describedabove in relation to FIG. 2. For example, as with the anatomy shown inFIG. 2, anatomy 504 associated with the operational scene ofconfiguration 500 may be internal anatomy within an internal operationalarea (e.g., an operational area under a patient's skin, which is notexplicitly depicted in FIG. 5). In other examples, anatomy 504 may beexternal anatomy and the operational area may not be completely enclosedby the patient's skin. However, it is noted that it may be possible, insuch examples, to provide physical far-range lighting to illuminate theoperational scene, rather than emulating such far-range lighting usingsystems and methods described herein. As such, system 400 may beunderstood to provide a particular benefit or advantage when theoperational area and operational scene are internal to the patient.

Image device 502 may be any suitable imaging device used for providingimaging device imagery of an operational scene. For example, imagingdevice 502 may be implemented by imaging device 200, described above, orany of the implementations thereof that have been described (e.g.,including endoscopes and other similar imaging tools as may beappropriate in different situations, for different types of operations,etc.). As shown, the portion of the operational scene captured byimaging device 502 (e.g., the captured portion of anatomy 504 and/orother objects present in the operational scene such as surgicalinstruments or the like that are not shown in FIG. 5) may be defined bya field of view 506 of imaging device 502. Field of view 506 may bedetermined based on various factors including, for example, the positionof imaging device 502 with respect to the operational scene, opticalcharacteristics associated with imaging device 502, intrinsic parametersof optics and/or image sensors included within imaging device 502, andso forth.

As shown within field of view 506 (as the dynamic field of view may beconstituted at a moment in time depicted in FIG. 5), different surfacepoints 508 (e.g., surface points 508-1 and 508-2) on the surface ofanatomy 504 within the operational scene may be different distances froma physical light source 510 from which close-range light originates toilluminate the operational scene. For example, as shown, physical lightsource 510 may be associated with (e.g., integrated into, implementedby, etc.) imaging device 502 and may be implemented at a distal end ofimaging device 502 in the same or a similar way as described above withilluminator 208 and illumination channel 214 in relation to FIG. 2.Close-range light illuminating the operational scene may thus originatefrom physical light source 510 as physical light source 510 is locatedat a first position that is a particular distance from various points inthe operational scene. For example, as shown, physical light source 510may be a distance D1 from surface point 508-1 and a distance D2 fromsurface point 508-2.

Even though the difference between D1 and D2 may be relatively small inquantity (e.g., a few centimeters or less), the difference may berelatively large as a total proportion of the distances. For instance,as shown, D2 is approximately twice as large as D1. Accordingly, due tothe close-range light originating from physical light source 510 and inaccordance with the inverse square law described above in relation toFIG. 3, light illuminating surface point 508-1 may be approximately fourtimes brighter than the light illuminating surface point 508-2. As aresult, the raw image of the operational scene captured by imagingdevice 502 may have a relatively large dynamic range with certain pixelsrepresentative of the region around surface point 508-1 being verybright and other pixels representative of the region around surfacepoint 508-2 being comparatively dim.

For reasons described above, it may be desirable to reduce this dynamicrange in such a way as to emulate far-range lighting for images of theoperational scene. For example, it may be desirable to process the rawimage so as to emulate lighting that, instead of originating at physicallight source 510, appears to originate at a virtual light source 512that is significantly more distant from the operational scene. As shown,for instance, virtual light source 512 may be simulated to be located ata position that is different from the position of physical light source510 and that is a greater distance from surface points 508 in theoperational scene. As with virtual light source 302, described above inrelation to FIGS. 3A and 3B, virtual light source 512 may be located ata virtual position in space associated with far range lighting, such asat a distance corresponding to position P4 in FIG. 3A, and/or at adistance where curve 310 has largely flattened out as shown in FIG. 3B.

As will be described in more detail below, far-range lighting may beemulated by determining, for each pixel captured in a raw image, afar-range lighting coefficient that is based on a ratio of the distanceof virtual light source 512 to the distance of physical light source510. Specifically, for example, a distance D3 between surface point508-1 and virtual light source 512 may be significantly greater thandistance D1 between surface point 508-1 and physical light source 510,and a far-range lighting coefficient associated with surface point 508-1may be based on a ratio of D3 to D1. Similarly, a distance D4 betweensurface point 508-2 and virtual light source 512 may be significantlygreater than distance D2 between surface point 508-2 and physical lightsource 510, and a far-range lighting coefficient associated with surfacepoint 508-2 may be based on a ratio of D4 to D2.

While the difference between D3 and D4 may be similar in quantity to thedifference between D1 and D2, the difference between D3 and D4 may bemuch smaller than the difference between D1 and D2 as a total proportionof the distances. For instance, while D2 is about 100% greater than D1,D4 may be only a negligible percentage greater than D3. Accordingly, dueto the far-range light virtually originating from virtual light source512 and in accordance with the inverse square law described above inrelation to FIG. 3, emulated light illuminating surface point 508-1 maybe approximately the same brightness as emulated light illuminatingsurface point 508-2 when the image is processed by system 400. As aresult, the processed image of the operational scene generated by system400 may have a relatively small dynamic range (e.g., smaller than thedynamic range of the raw image captured by imaging device 502) withpixels representative of regions around both surface points 508 beingapproximately the same brightness.

Accordingly, FIG. 5 illustrates a primary aim of system 400, which is togenerate a processed image that is based on a raw image captured byimaging device 502 of the operational scene, but that emulates far-rangelighting that virtually originates at virtual light source 512 in placeof the close-range lighting that physically illuminates the operationalscene. As used herein, the terms “close-range light” and “far-rangelight,” as well as related terms (e.g., referring to “lighting” ratherthan “light,” etc.), are relative terms best understood in relation toone another and in the context of a particular imaging configuration.

For example, close-range light may refer to light that originates closeenough to a point on a surface that other surrounding points at slightlydifferent distances may be illuminated with a significantly differentbrightness as a result of the inverse square law described above. In thecontext of operations performed in an internal operational area (e.g.,minimally invasive surgical procedures) described in various examplesherein, for example, close-range light may refer to light originatingonly a few centimeters (e.g., less than 10 cm, less than 20 cm, etc.)from an operational scene being imaged.

Far-range light may refer to any light that is not close-range light.For example, far-range light may refer to light that originates farenough from the point on the surface that other surrounding points atslightly different distances may be illuminated with practically thesame brightness as a result of the inverse square law. In the context ofinternal operations described herein, for example, far-range light mayrefer to light originating more than a few centimeters (e.g., more than10 cm, more than 20 cm, etc.) from the operational scene being imaged.In certain examples, far-range light may originate at a predefined oruser-selected distance that is preferred by a particular surgical teamor that simulates a particular lighting scenario such as an open surgerylighting scenario. In other examples, far-range light may originate atgreat distances so as to emulate, for example, sunlight or the like.

Once system 400 has generated the processed image to emulate far-rangelighting originating from virtual light source 512, system 400 mayprovide the processed image to a display screen 514 such as displaymonitor 114 of auxiliary system 106, a display screen included withinuser control system 104 in surgical system 100, or another suitabledisplay screen. In this way, certain or all surgical team members 110may view an attractive, minimally saturated, and uniformly bright imageof the operational scene as they perform the operation.

While system 400 is illustrated as being communicatively coupled toimaging device 502 and display screen 514 in configuration 500, it willbe understood that lighting emulation systems described herein, such assystem 400, may be implemented by any devices and/or in any manner asmay serve a particular implementation. For instance, system 400 may, incertain examples, be implemented by or otherwise integrated with imagingdevice 502. In other examples, system 400 may be implemented orintegrated with auxiliary system 106, user control system 104, or anyother suitable component of surgical system 100 or other computingdevice as may serve a particular embodiment.

Additionally, in certain examples, it will be understood that a lightingemulation system may include more than just the computing resourcesdescribed above (e.g., processors, memories, etc.). For instance, oneexemplary implementation of a lighting emulation system may include animaging device configured to be located at an operational area thatincludes an operational scene; a physical light source associated withthe imaging device and configured to illuminate the operational scene;an image sensor included within the imaging device and configured tocapture an image depicting the operational scene as illuminated byclose-range light originating from the physical light source; a displayscreen configured to present images captured by the imaging device; aprocessor communicatively coupled to the imaging device; and a memorycommunicatively coupled to the processor and storing instructions thatare executable by the processor to direct the processor to perform anyof the operations described herein. For example, the processor, underdirection of the instructions stored by the memory, may direct theimaging device to capture, by way of the image sensor, an imagedepicting the operational scene as illuminated by the close-range lightoriginating from physical light source; generate, based on the imagedepicting the operational scene, a depth map of the operational scenethat includes depth data corresponding to each pixel in the image;determine, based on the depth map, a far-range lighting coefficient foreach pixel in the image (e.g., in which the (ar-range lightingcoefficient for each respective pixel is determined based on thecorresponding depth data included in the depth map for that respectivepixel); generate, based on the image and the far-range lightingcoefficient for each pixel in the image, a processed image depicting theoperational scene as being illuminated by far-range lighting; andprovide the processed image for presentation on the display screen.

Various exemplary details of how system 400 may perform the emulating offar-range lighting for operational scenes illuminated by close-rangelight within configurations such as configuration 500 will now bedescribed. Specifically, FIGS. 6 and 7 illustrate exemplary brightnessvalues of certain pixels of different exemplary images that depict anoperational scene illuminated by close-range light. In particular, aswill be described, FIG. 6 illustrates exemplary brightness values for animage captured using one exposure algorithm, while FIG. 7 illustratescorresponding brightness values for an exemplary image captured using adifferent exposure algorithm, FIG. 8 illustrates exemplary depth datastored in an exemplary depth map that may be generated, received, orotherwise accessed by system 400. Based on the images and depth mapillustrated in FIGS. 6 through 8, FIG. 9 illustrates an exemplarydataflow for operations performed by an exemplary implementation ofsystem 400. Ultimately, the dataflow of FIG. 9 leads system 400 togenerate a processed image. Accordingly, FIG. 10 shows exemplarybrightness values of certain pixels of the processed image that resultsfrom processing a raw image in accordance with the dataflow of FIG. 9.

Referring now to FIG. 6, brightness values of certain pixels included inan image 602 are shown. As used herein, a “picture element” (“pixel” forshort) may refer to any element of an image or depth map as may serve aparticular implementation. For instance, in certain examples, a pixelmay refer to a single point included within an image or a depth map. Inother examples, a pixel may refer to a grouping of related points withinthe image or depth map. As one example, a single color pixel within animage may include a red point, a green point, and a blue point. Asanother example, a pixel within an image or depth map may include agrouping of related points within an area of the image or depth map. Forinstance, if multiple points in a region of an image are determined tohave a same depth or to correspond with a single pixel of acorresponding depth map, all the points may be processed together as asingle picture element. By processing groupings of related pointstogether, rather than individually, in this way, system 400 may processimages and/or depth maps more effectively and/or efficiently.Additionally, grouping points together into processable picture elementsin this way may allow system 400 to operate with a relatively highdegree of flexibility so as to not require, for example, that each pointmaking up a particular image correspond one-to-one with a particularpoint making up a corresponding depth map. For instance, by groupingmultiple image points into each of the “pixels” processed in the waysdescribed herein, system 400 may flexibly correlate a relatively lowresolution depth map with an image having a higher resolution (e.g.,possibly after performing post-filtering to, for example, remove noisein the depth data, smooth the depth map, fill holes, etc.).

Image 602 will be understood to be a “raw” image (i.e., an imagecaptured by imaging device 502 and which has not yet been processed bysystem 400 to alter the brightness of any pixel). For example, image 602may be captured by either a right-side or a left-side image sensor ifimaging device 502 is a stereoscopic imaging device, or by a monoscopicimage sensor if imaging device 502 is a monoscopic imaging device.

As illustrated by a key 604 under image 602, different upper-caseletters “A” through “Z” are used to illustrate different brightnessvalues for each pixel in image 602. The pixels are represented by thesmall squares making up image 602. As indicated by key 604, pixelsrepresenting relatively dim (“Less Bright”) points of an operationalscene are marked with earlier letters in the alphabet (e.g., “A,” “B,”“C,” etc.), with pixels representing areas so dim that no appreciableamount of light is detected by the image sensor being depicted as blacksquares. In contrast, pixels representing relatively bright (“Brighter”)points of the operational scene are marked with later letters in thealphabet (e.g., “Z,” “Y,” “X,” etc.), with pixels representing areas sobright the image sensor saturates being depicted as white squares.

Also shown in key 604 are pixels outlined by dotted, rather than solid,lines. These pixels will be understood to have brightness values thatare similar to the values of the pixels outlined with solid lines, eventhough these values are not explicitly specified in FIG. 6 for clarityof illustration. Instead, brightness values are specified only for a fewselect groups of pixels within image 602 to illustrate principlesdescribed below. Specifically, as shown, a pixel group 606 includespixels with a mid-range of brightness (e.g., values from “L” to “O”), apixel group 608 and a pixel group 610 include pixels that are so brightas to be saturated (e.g., appearing as a glare or “hot spot” in theimage), and a pixel group 612 includes pixels that are so dim as to notinclude any information.

The brightness values illustrated in FIG. 6 may represent the brightnesscaptured when the imaging device uses a conventional auto-exposurealgorithm to make the majority of pixels in image 602 take on abrightness value as close to a mid-range value as possible. Using thisauto-exposure algorithm, pixels in pixel group 606 may be captured witha brightness level that looks attractive to viewers of image 602 (i.e.,a brightness level that is not so bright as to appear washed out whilealso being not so dim as to make it difficult for a viewer to perceivedetail in the image). While most pixels in image 602 are not specifiedfor clarity of illustration, it will be understood that the conventionalauto-exposure algorithm may also make many other pixels in image 602look similarly attractive to viewers. However, as shown, it will also beunderstood that, regardless of what exposure algorithm is used, certainpixels may be brighter or dimmer than others for various reasonsincluding reasons that may be unrelated to depth variations in theoperational scene. For example, pixels in pixel group 610 may representan area of the operational scene in which light is directly reflected atthe imaging device to create a glare that would saturate these pixelsusing any practical exposure algorithm. Similarly, pixels in pixel group612 may represent an area (e.g., a dark crevice, etc.) of theoperational scene in which light is absorbed so as not to reflect anyamount of light detectable using any practical exposure algorithm.

Pixel group 608, however, may be in a different situation than pixelgroups 610 and 612. While pixel groups 610 and 612 may practicallyalways fall to one brightness extreme or the other by the nature of thecontent being depicted by these pixel groups, pixel group 608 may onlybe saturated as a result of the exposure time (e.g., the exposure timeimplemented by the auto-exposure algorithm) being too long for thatportion of image 602. Accordingly, in the case of pixel group 608, ifthe exposure time were to be shortened, it may be possible for thepixels in pixel group 608 to provide useful information rather than tosaturate as shown in image 602.

To illustrate, FIG. 7 shows an image 702 that depicts the same content(e.g., the same objects within the same operational scene) as image 602,but is captured using an exposure algorithm that is configured to exposethe image for a shorter time than the exposure algorithm associated withimage 602. For example, prior to accessing image 702 depicting theoperational scene, system 400 may direct the imaging device to captureimage 702 using an exposure algorithm configured to underexpose theimage to minimize saturation of the pixels in the image. While theexposure algorithm of FIG. 6 may represent a conventional auto-exposurealgorithm configured to optimize the brightness of as many pixels in theimage as possible, the exposure algorithm of FIG. 7 may, by design,underexpose the image so that many pixels may appear darker than may bepreferred by typical viewers. The advantage of this underexposure,however, is that the exposure algorithm of FIG. 7 may capture moreinformation than the exposure algorithm of FIG. 6, thereby allowing morepixels to be meaningfully processed and adjusted by system 400 (afterwhich the brightness may be readjusted to a more preferred level, aswill be described in more detail below).

To illustrate, for example, meaningful brightness values of the pixelsin pixel group 606 are still represented in image 702, even though thesepixels may appear relatively dark to a viewer viewing image 702.Specifically, rather than the mid-range brightness values of thesepixels shown in image 602 (e.g., from “L” to “0”), the pixels havedimmer brightness values in image 702 (e.g., from “D” to “G”). However,whereas the pixels in pixel group 608 were saturated in image 602 so asto not provide any useful information, these pixels represent meaningfulbrightness values (albeit very bright values from to “Z”) in image 702.As shown in FIG. 7, and for the reasons described above, other pixelssuch as the pixels in pixel group 610 may still be saturated whilepixels in pixel group 612 may still be too dark to provide usefulinformation.

The exposure algorithm used to capture image 702 may be implementedusing any criteria as may serve a particular implementation. Forexample, in some implementations, the exposure algorithm may beconfigured to underexpose the image based on the brightest areas beingcaptured (e.g., to minimize saturated pixels). In other implementations,the exposure algorithm may be configured to maximize the amount ofmeaningful data that is captured by underexposing the image but only tothe extent that more brightness data is captured rather than less data(which could happen, for example, if too many relatively dim pixels wereunderexposed so as to not provide any useful brightness data). In thisway, the exposure algorithm may account for pixels near both extremes ofbrightness to gather as much brightness information as possible giventhe dynamic range of the imaging device.

As mentioned above, along with accessing an image (e.g., such as image602 or image 702) from imaging device 502, system 400 may further accessa depth map. The depth map may be generated in any suitable mannerand/or by any device as may serve a particular implementation. Forexample, in certain implementations, the depth map may be captured by adepth imaging tool separate from imaging device 502 or integrated withimaging device 502. Such a depth imaging tool may use a time-of-flightdepth capture technique, for example, or any other depth capturetechnique as may serve a particular implementation.

In other examples, no specialized depth imaging tool may be necessary togenerate the depth map, but, rather, the depth map may be generatedbased on a plurality of images captured by imaging device 502 fromdifferent vantage points (e.g., stereoscopic vantage points). Forinstance, system 400 may access the depth map of the operational sceneby accessing a first image and a second image from imaging device 502,where the first and second images depict the operational scene fromstereoscopic vantage points having a predetermined spatial relationship.System 400 may further compare the first image to the second image andgenerate the depth map based on the comparison of the first image to thesecond image.

FIG. 8 shows a depth map 802 that includes depth data corresponding toeach of the pixels of images 602 and 702 described above. Similar toimages 602 and 702, depth map 802 is associated with a key 804 underdepth map 802 that sets forth the notation used in FIG. 8. Specifically,as illustrated by key 804, different lower-case letters (e.g., “a”through “z”) are used to illustrate different depth data values for eachpixel in depth map 802. As with images 602 and 702, the pixels of depthmap 802 are represented by the small squares making up depth map 802. Asindicated by key 804, pixels representing relatively distant (“FartherAway”) points of an operational scene are marked with earlier letters inthe alphabet (e.g., “a,” “b,” “c,” etc.). In contrast, pixelsrepresenting relatively near (“Closer”) points of the operational sceneare marked with later letters in the alphabet (e.g., “z,” “y,” “x,”etc.).

Also shown in key 804 are pixels outlined by dotted, rather than solid,lines. As with similar pixels in images 602 and 702, these pixels willbe understood to have values (depth values, in this case) that aresimilar to the values of the pixels outlined with solid lines, eventhough the values are not explicitly specified in FIG. 8 for clarity ofillustration. Instead, depth values are specified only for the sameselect groups of pixels explicitly shown in images 602 and 702 (i.e.,pixel groups 606, 608, 610, and 612).

As shown, pixel group 606 in depth map 802 may be relatively far awayfrom the vantage point of the tool capturing depth map 802 (e.g.,imaging device 502). Specifically, as shown, pixel group 606 includesdepth values “a” through “c”. At least partially because of this depth,and as shown in FIGS. 6 and 7, pixels in pixel group 606 may berelatively dim compared to other pixels in the images such as those inpixel group 608. In contrast, pixel group 608 in depth map 802 may berelatively near the vantage point of the tool capturing depth map 802.Specifically, as shown, pixel group 608 includes depth values “s”through “u”. At least partially because of this depth, and as shown inFIGS. 6 and 7, pixels in pixel group 608 may be relatively brightcompared to other pixels in the images such as those in pixel group 606.Pixels in pixel groups 610 and 612 also are shown to be associated withrespective depth data, although it will be appreciated that therespective brightness extremes of these pixel groups appear to have moreto do with the nature of the content being depicted (e.g., glares, darkcrevices, etc.) than the depth of the content being depicted (which, asshown, may not be particularly extreme).

Next to depth map 802 in FIG. 8, an analysis 806 of two images 702(i.e., images 702-1 and 702-2) is shown to be the source from whichdepth map 802 is derived. Images 702-1 and 702-2 will be understood tobe the same or similar to image 702 illustrated in FIG. 7 in that anexposure algorithm configured to minimize pixel saturation and/ormaximize captured brightness information may be used to generate theseimages. However, it will also be understood that images 702-1 and 702-2are not identical, but, rather, are stereoscopically distinct from oneanother. Accordingly, as mentioned above, depth map 802 may be generatedby comparing images 702-1 and 702-2 and/or otherwise processing theimages using any type of stereoscopic depth derivation algorithm as mayserve a particular implementation.

Images 702 in FIG. 8 may be stereoscopic images that are captured fromdifferent, but related, vantage points to allow captured features in theimages to be correlated and depth to be determined based on a knownrelationship between the vantage points. The stereoscopic images may becaptured in any manner and by any device (e.g., any imaging device withany configuration of one or more image sensors) as may serve aparticular implementation. For instance, in some implementations,imaging device 502 may be a stereoscopic imaging device such as imagingdevice 200 that includes a first image sensor and a second image sensorthat is stereoscopic with the first image sensor such that the first andsecond image sensors together capture sets of synchronized stereoscopicimages (i.e., stereoscopic images captured at approximately the samemoment in time). In these implementations, the images 702-1 and 702-2may thus be included within a particular set of synchronizedstereoscopic images captured by the first and second image sensorswithin the stereoscopic implementation of imaging device 502.

In other implementations, imaging device 502 does not necessarily needto be a stereoscopic imaging device. For instance, image 702-1 may becaptured at a first time by an image sensor included within a monoscopicor stereoscopic implementation of imaging device 502, and image 702-2may be captured at a second time by the same image sensor includedwithin the monoscopic or stereoscopic implementation of imaging device502, where the second time is distinct from the first time. For example,as the imaging device 502 is moved within the operational area tocapture different portions of the operational scene, different imagesmay be captured from slightly different vantage points, and these imagesmay be used as images 702-1 and 702-2 from which depth map 802 isderived by way of analysis 806. For example, the relationship betweenthe vantage points in this example may be determined based on kinematicdata or in any other suitable manner.

As described above, once data access facility 402 in system 400 accessesan image such as image 702 and a depth map such as depth map 802, dataanalysis facility 404 and data generation facility 406 may process theimage and the depth map to generate a processed version of the image andto provide the processed image for presentation on a display screen.

To illustrate, FIG. 9 shows an exemplary dataflow 900 used by facilities404 and 406 within system 400 as the processed image is generated andprovided. Starting at the left and moving toward the right, dataflow 900shows that accessed depth map 802 may be used to identify detected depthdata (e.g., “Actual Distances”) of various points included in theoperational scene. Specifically, dataflow 900 depicts the flow for dataassociated with two particular points of an operational scene: surfacepoints 508-1 and 508-2 illustrated above in relation to FIG. 5. Asshown, an actual distance 902-1 between surface point 508-1 and aphysical light source illuminating the operational scene (e.g., physicallight source 510) is identified based on depth data included withindepth map 802, and an actual distance 902-2 between surface point 508-2and the physical light source is also identified based on depth datafrom depth map 802. More particularly, as annotated in FIG. 9, actualdistance 902-1 will be understood to correspond to distance D1illustrated in FIG. 5, while actual distance 902-2 will be understood tocorrespond to distance D2. While data for surface points 508 is shownexclusively in dataflow 900 for illustrative clarity, it will beunderstood that surface points 508 are exemplary only, and that system400 may perform similar operations for every surface point depicted byevery pixel in depth map 802 and image 702, and not just those pixelsdepicting surface points 508-1 and 508-2.

Along with actual distances 902, dataflow 900 also shows targetdistances 904 corresponding to surface points 508 (i.e., target distance904-1 corresponding to surface point 508-1 and target distance 904-2corresponding to surface point 508-2). Target distances 904 representrespective distances from surface points 508 to a position of virtuallight source 512, which, as described above in relation to FIG. 5,virtually originates far-range light that system 400 is configured toemulate. Accordingly, as annotated, target distance 904-1 will beunderstood to correspond to distance D3 illustrated in FIG. 5, whiletarget distance 904-2 will be understood to correspond to distance D4.

Target distances 904 (and, thus, the virtual position of virtual lightsource 512 that the target distances serve to define) may be accessed ordetermined by system 400 in any manner as may serve a particularimplementation. For example, a predefined position a certain distancefrom the operational scene may be used to define target distances 904.In one embodiment, for instance, target distances 904 may be two metersfrom the operational scene or some other specific value that may serve aparticular implementation. As another example, target distances 904 maybe set to be a predefined (e.g., statically set or user selected)distance configured to emulate a certain lighting environment. Forinstance, as mentioned above, target distances 904 may be defined tovirtually locate virtual light source 512 at a position emulating aposition of a light source that would be used in an open surgery.

In other implementations, target distances 904 may not be predefined,but, rather, may be dynamically configurable by a user such as a surgeonperforming an operation or a surgical team member associated with theoperation. Specifically, for instance, system 400 may be configured toprompt a user of system 400 to provide input, and to receive, from theuser, user input representative of a user selection of the position atwhich the virtual light source is simulated to be located. System 400may then define the virtual light source position based on the userinput by, for instance, defining target distances 904 in accordance withthe position selected by the user and the depth data for each pointprovided by depth map 802.

In prompting the user to provide the user input representative of theuser selection of the position at which the virtual light source is tobe simulated to be located, system 400 may use any user interfaces asmay serve a particular type of user (e.g., a surgeon using user controlsystem 104, another surgical team member using auxiliary system 106,etc.) in a particular implementation. For example, in some examples, aphysical or graphical slider input tool (e.g., implemented by a touchscreen or other suitable input mechanism) may be provided forpresentation within a user interface presented to the user of thesystem. The slider input tool may be configured to facilitate the userselection of the virtual light source position, and system 400 maydefine the position at which the virtual light source is simulated to belocated based on a setting of the slider input tool selected by theuser. In some examples, the slider input tool may be presented to theuser in terms of a position of a virtual light source (e.g., by beinglabeled “Virtual Light Source Distance,” or the like), while, in otherexamples, the details of what is being simulated may be abstracted awayfrom the user to some extent (e.g., by labeling the slider input tool“Dynamic Contrast,” or the like).

A computation 906-1 is shown in dataflow 900 to receive as input theactual distances 902 and target distances 904, and to use this data tocompute respective far-range lighting coefficients 908 for each point(e.g., a far-range lighting coefficient 908-1 for surface point 508-1based on distances 902-1 and 904-1, a far-range lighting coefficient908-2 based on distances 902-2 and 904-2, etc.). As such, computation906-1 may be performed by data analysis facility 404 to perform theoperations described above in relation to FIG. 4.

Each far-range lighting coefficient 908 may be implemented as acustomized multiplier for a particular pixel in a raw image such asimage 702. Each far-range lighting coefficient takes into account 1) adistance from a physical light source to the point depicted by thepixel, and 2) a distance from a virtual light source to the pointdepicted by the pixel, such that, when multiplied by a raw brightnessvalue captured for the pixel, the far-range lighting coefficient adjuststhe brightness of the pixel to emulate the point depicted by the pixelas being illuminated by the far-range light of the virtual light source.Because far-range lighting coefficients do not arbitrarily brighten dimpixels and/or dim bright pixels, but, rather, facilitate an emulation ofhow each pixel would depict its respective surface point if theoperational scene were illuminated by the virtual light source,far-range lighting coefficients facilitate the generation of a processedimage that not only has high brightness uniformity, but also that looksattractive and realistic, rather than distractingly artificial orunnatural.

Computation 906-1 may determine far-range lighting coefficients 908 inany manner as may serve a particular implementation. For example, asmentioned above, system 400 may determine each far-range lightingcoefficient for each surface point based on a ratio of the targetdistance 904 for that point to the actual distance 902 for that point.Specifically, a far-range lighting coefficient for a particular surfacepoint 508 may be computed based on Equation 2, set forth below.

$\begin{matrix}{{{FRLC}( {D_{1},D_{2}} )} \cong \frac{1}{( \frac{D_{2}}{D_{1}} )^{2}}} & ( {{Equation}\mspace{14mu} 2} )\end{matrix}$

In Equation 2, FRLC(D₁,D₂) represents a far-range lighting coefficientfor a particular point (e.g., one of surface points 508) that is adistance D1 from a physical light source and a distance D2 from avirtual light source that is to be simulated to be illuminating thepoint. Taking surface point 508-1 as an example, D1 in Equation 2corresponds to actual distance 902-1 (i.e., D1 in FIG. 5) and D2 inEquation 2 corresponds to target distance 904-1 (i.e., D3 in FIG. 5).Using surface point 508-2 as an additional example, D1 in Equation 2corresponds to actual distance 902-2 (i.e., D2 in FIG. 5) and D2 inEquation 2 corresponds to target distance 904-2 (i.e., D4 in FIG. 5).Computation 906-1 may compute respective far-range lighting coefficients508 for each point represented in depth map 802 and/or image 702 in thisway.

It will be noted, as indicated in Equation 2, that the value computed asthe inverse of the square of the ratio of D2 to D1 may be approximately,but not necessarily exactly, equal to an ideal far-range lightingcoefficient that would precisely emulate far-range lighting from thevirtual light source. This is because, as described above, the physicallight source closely approximates a point light source but is not a truemathematical point light source. Additionally, the point being captureddoes not actually originate close-range light, but rather reflectsclose-range light originated by the physical light source. Nevertheless,far-range lighting coefficients computed using Equation 2 provide goodapproximations to ideal far-range lighting coefficients and mayadvantageously be computed with relative efficiency in comparison tofar-range lighting coefficients computed based on more complex andcomprehensive models. It will be understood, however, that inimplementations where more precision is desired and computationresources are available to support such precision, more precise modelsand complex equations may be used in place of Equation 2.

In computation 906-2, far-range lighting coefficients 908-1 and 908-2may be combined (e.g., multiplied) with respective brightness values910-1 and 910-2 accessed based on a captured raw image (e.g., image702). Because, as described above, image 702 may be captured using anexposure algorithm configured to minimize saturated pixels and/or tomaximize the amount of brightness information captured, all or nearlyall of the computed far-range lighting coefficients 908 may be combinedwith captured brightness values representing meaningful data (e.g.,rather than representing a saturated or dark pixel). Computation 906-2may be performed by data generation facility 406 in system 400.Specifically, computation 906-2 may be performed sequentially or inparallel for each respective pixel (x, y) included in image 702 bymultiplying a respective brightness value 910 of each respective pixelin image 702 by the computed far-range lighting coefficient 908determined for that respective pixel in accordance with Equation 3 setforth below.

B _(P)(x,y)=B _(U) B _(R)(x,y)FRLC(D ₁(x,y)D ₂(x,y))  (Equation 3)

In Equation 3, B_(P)(x,y) represents the processed brightness of a pixel(x,y) in image 702. More specifically, the brightness value ofB_(P)(x,y) is the brightness of that pixel when the far-range lightvirtually originating from the virtual light source is emulated in thedepiction of the point depicted by pixel (x,y). FRLC(D₁(x,y), D₂(x,y))represents a far-range lighting coefficient 908 for pixel (x,y) computedby computation 906-1 in accordance with Equation 2, as described above.As shown, this value is multiplied by B_(P)(x,y), which represents theraw brightness value 910 of pixel (x,y) based on image 702. Bymultiplying each raw brightness value 910 by a respective far-rangelighting coefficient 908 in this way, the resultant processed brightnessof the pixels may be more uniform than the raw brightness of the pixelsas captured in image 702. As such, if image 702 as captured by theimaging device is associated with a first dynamic range, a processedimage including pixels adjusted in accordance with Equation 3 will beassociated with a second dynamic range that is less than the firstdynamic range.

Because the virtual light source is positioned farther away from thesurface point being depicted than the physical light source (i.e.,because D2 is greater than D1), it will be recognized that far-rangelighting coefficients 908 computed using Equation 2 are greater than 0and less than 1. For example, a far-range lighting coefficient of 0would represent a virtual light source that is infinitely far away,while a far-range lighting coefficient of 1 would represent a virtuallight source positioned exactly at the same position as the physicallight source. When such far-range lighting coefficients 908 aremultiplied by brightness values 910, the resultant product is thusattenuated from (i.e., less than) the raw brightness value 910.Moreover, as described above in relation to FIG. 7, raw brightnessvalues 910 associated with image 702 may already be intentionallyunderexposed to some degree so as to minimize saturated pixels in theraw image and maximize the captured information. Consequently, while theproduct of respective far-range lighting coefficients 908 and brightnessvalues 910 may be relatively uniform for various pixels and may providea relatively small dynamic range, the resultant image may appear verydim to a user viewing the image.

To remedy this, Equation 3 includes an additional factor B_(U)representative of a universal brightness adjustment that may be made tobrighten up all the pixels in the resultant image by the same amount. Assuch, B_(U) will be understood to be a constant, rather than a functionof pixel (x,y), as with other factors included in Equation 3.

Dataflow 900 illustrates a universal brightness adjustment 914 that maybe performed on the respective products of far-range lightingcoefficients 908 and brightness values 910 multiplied in computation906-2. As mentioned above and as shown by B_(U) in Equation 3, universalbrightness adjustment 914 may increase the brightness of each pixelequally so as to generate a final processed image 916 that is at anattractive level of brightness for presentation to viewers, but thatdoes not affect the reduced dynamic range (i.e., the more uniformbrightness) achieved by computation 906-2. For example, system 400(e.g., data generation facility 406) may perform universal brightnessadjustment 914 on each pixel of the processed image subsequent togenerating the processed image (i.e., subsequent to performingcomputation 906-2) and prior to providing processed image 916 forpresentation on a display screen, as shown. Universal brightnessadjustment may be performed by multiplying a brightness value of eachpixel in the processed image by a universal lighting coefficient (e.g.,B_(U) in Equation 3).

The amount by which each pixel is universally brightened (i.e., themagnitude of B_(U) in Equation 3) may be determined in any manner as mayserve a particular implementation. For instance, the brightening factormay be determined using an autoexposure algorithm such as a conventionalautoexposure algorithm used to generate image 602 in FIG. 6. In otherexamples, another slider input tool similar to the one described above;or another suitable user input mechanism, may be provided in a userinterface to allow users to manually select a desired brightness level.

Additionally, in some examples, universal brightness adjustment 914 maybe proceeded or succeeded by one or more other adjustment stages (notexplicitly shown in FIG. 9) that are configured to adjust the brightnessof each pixel in a localized, non-universal manner as part of generatinga processed image 916. As one example, once the brightness of each pixelhas been adjusted to simulate, for instance, an autoexposed imageilluminated with far-range lighting from a virtual position removed fromthe operational scene (e.g., the position of virtual light source 512),system 400 may further adjust the brightness of certain pixels in orderto simulate shadowing effects as illuminated by the far-range lightingat the virtual position. System 400 may adjust brightness in localizedareas of the image to simulate such shadowing effects in any suitableway. For example, system 400 may adjust the brightness values by usingsimulated ray-casting techniques to calculate where shadows should, andshould not, be present in the operational scene based on the simulatedposition of the virtual light source. Adding and/or adjusting suchshadows may provide an additional element of realism for processed image916, making the operational scene in the processed image more genuinelyappear to be illuminated by the far-range lighting at the virtualposition outside the operational area.

While a single image (e.g., image 702) captured at a single point intime has been described in many of the examples up to this point, itwill be understood that image 702 depicting the operational scene may beincluded within a sequence of images depicting the operational scene.Specifically, for example, an image sequence (e.g., a video stream)composed of a large number of sequential images may be captured by theimage sensor of imaging device 502 and, as such, there may be variousprior images captured within the image sequence before image 702. Insome examples, similar processing that has been performed for such priorimages may facilitate (e.g., provide processing “shortcuts”) for theprocessing of image 702 and/or depth map 802 as shown in dataflow 900.For example, system 400 may be configured to access one or more priorimages and/or depth maps that have already been processed beforeaccessing image 702 and/or depth map 802, and may perform certainoperations (e.g., the accessing of depth map 802, the determination offar-range lighting coefficients 908 for each pixel in the image, etc.)based on the accessed prior image and/or depth map. For example,far-range lighting coefficients 908 may not tend to change significantlyfrom frame to frame in an image sequence, so system 400 may not updateall of the far-range lighting coefficients for each frame, but, rather,may access (e.g., from storage facility 408) the far-range lightingcoefficients calculated for a previous image and only update thefar-range lighting coefficients periodically (e.g., every other frame,every ten frames, every second, every minute, etc.).

As shown in FIG. 9, at the end of dataflow 900, processed image 916 isgenerated that may be provided for display on any of the display screensdescribed herein. To illustrate, FIG. 10 shows exemplary brightnessvalues of certain pixels of processed image 916. As shown in FIG. 10,processed image 916 is similar to images 602 and 702 and uses the samekey (i.e., key 604) to notate brightness values of different pixels inthe pixel groups. Like images 602 and 702, pixel group 610 in processedimage 916 includes the saturated pixels and pixel group 612 includes thedark pixels that are not altered by the processing of system 400 for thereasons described above. However, as illustrated by pixel groups 606 and608, other (non-extreme) pixels in image 916 have been altered to becomemore uniform, such that processed image 916 has a lower dynamic rangethat will make the image more attractive to present on the displayscreen, easier to view and analyze in detail by viewers, and so forth.Specifically, as shown, pixels in pixel group 606 have been brightenedas compared to the raw values shown in this pixel group in image 702,while pixels in pixel group 608 have been dimmed as compared to the rawvalues shown in this pixel group in image 702. Accordingly, rather thana dynamic range from “D” to “Z,” as shown in image 702, the dynamicrange of brightness has been reduced in processed image 916 to a dynamicrange from “F” to “5”.

FIG. 11 illustrates an exemplary method 1100 for emulating far-rangelighting for an operational scene illuminated by close-range light.While FIG. 11 illustrates exemplary operations according to oneembodiment, other embodiments may omit, add to, reorder, and/or modifyany of the operations shown in FIG. 11. One or more of the operationsshown in FIG. 11 may be performed by a lighting emulation system such assystem 400, any components included therein, and/or any implementationthereof.

In operation 1102, a lighting emulation system may access an imagedepicting an operational scene illuminated by close-range light. Forexample, the image may be captured by an image sensor included within animaging device located at an operational area that includes theoperational scene. Operation 1102 may be performed in any of the waysdescribed herein.

In operation 1104, the lighting emulation system may access a depth mapof the operational scene. The depth map accessed in operation 1104 mayinclude depth data corresponding to each pixel in the image accessed inoperation 1102. Operation 1104 may be performed in any of the waysdescribed herein.

In operation 1106, the lighting emulation system may determine afar-range lighting coefficient for each pixel in the image accessed inoperation 1102. For example, the lighting emulation system may determinethe far-range lighting coefficient based on the depth map accessed inoperation 1104. In some examples, the determining of the far-rangelighting coefficient for each respective pixel in operation 1106 may beperformed based on the corresponding depth data included in the depthmap for that respective pixel. Operation 1106 may be performed in any ofthe ways described herein.

In operation 1108, the lighting emulation system may generate aprocessed image depicting the operational scene as being illuminated byfar-range lighting. For example, the lighting emulation system maygenerate the processed image based on the image accessed in operation1102 and further based on the far-range lighting coefficient for eachpixel in the image determined in operation 1106. Operation 1108 may beperformed in any of the ways described herein.

In operation 1110, the lighting emulation system may provide theprocessed image for presentation on a display screen. Operation 1110 maybe performed in any of the ways described herein.

In certain embodiments, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices. In general, a processor (e.g., a microprocessor,etc.) receives instructions, from a non-transitory computer-readablemedium, (e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein. Such instructions may be stored and/or transmittedusing any of a variety of known computer-readable media.

A computer-readable medium (also referred to as a processor-readablemedium) includes any non-transitory medium that participates inproviding data (e.g., instructions) that may be read by a computer(e.g., by a processor of a computer). Such a medium may take many forms,including, but not limited to, non-volatile media, and/or volatilemedia. Non-volatile media may include, for example, optical or magneticdisks and other persistent memory. Volatile media may include, forexample, dynamic random access memory (“DRAM”), which typicallyconstitutes a main memory. Common forms of computer-readable mediainclude, for example, a disk, hard disk, magnetic tape, any othermagnetic medium, a compact disc read-only memory (“CD-ROM”), a digitalvideo disc (“DVD”), any other optical medium, random access memory(“RAM”), programmable read-only memory (“PROM”), electrically erasableprogrammable read-only memory (“EPROM”), FLASH-EEPROM, any other memorychip or cartridge, or any other tangible medium from which a computercan read.

FIG. 12 illustrates an exemplary computing device 1200 that may bespecifically configured to perform one or more of the processesdescribed herein. As shown in FIG. 12, computing device 1200 may includea communication interface 1202, a processor 1204, a storage device 1206,and an input/output (“I/O”) module 1208 communicatively connected via acommunication infrastructure 1210. While an exemplary computing device1200 is shown in FIG. 12, the components illustrated in FIG. 12 are notintended to be limiting. Additional or alternative components may beused in other embodiments. Components of computing device 1200 shown inFIG. 12 will now be described in additional detail.

Communication interface 1202 may be configured to communicate with oneor more computing devices. Examples of communication interface 1202include, without limitation, a wired network interface (such as anetwork interface card), a wireless network interface (such as awireless network interface card), a modem, an audio/video connection,and any other suitable interface.

Processor 1204 generally represents any type or form of processing unitcapable of processing data or interpreting, executing, and/or directingexecution of one or more of the instructions, processes, and/oroperations described herein. Processor 1204 may direct execution ofoperations in accordance with one or more applications 1212 or othercomputer-executable instructions such as may be stored in storage device1206 or another computer-readable medium.

Storage device 1206 may include one or more data storage media, devices,or configurations and may employ any type, form, and combination of datastorage media and/or device. For example, storage device 1206 mayinclude, but is not limited to, a hard drive, network drive, flashdrive, magnetic disc, optical disc, RAM, dynamic RAM, other non-volatileand/or volatile data storage units, or a combination or sub-combinationthereof. Electronic data, including data described herein, may betemporarily and/or permanently stored in storage device 1206. Forexample, data representative of one or more executable applications 1212configured to direct processor 1204 to perform any of the operationsdescribed herein may be stored within storage device 1206. In someexamples, data may be arranged in one or more databases residing withinstorage device 1206.

I/O module 1208 may include one or more I/O modules configured toreceive user input and provide user output. One or more I/O modules maybe used to receive input for a single virtual reality experience. I/Omodule 1208 may include any hardware, firmware, software, or combinationthereof supportive of input and output capabilities. For example, I/Omodule 1208 may include hardware and/or software for capturing userinput, including, but not limited to, a keyboard or keypad, atouchscreen component (e.g., touchscreen display), a receiver (e.g., anRF or infrared receiver), motion sensors, and/or one or more inputbuttons.

I/O module 1208 may include one or more devices for presenting output toa user, including, but not limited to, a graphics engine, a display(e.g., a display screen), one or more output drivers (e.g., displaydrivers), one or more audio speakers, and one or more audio drivers. Incertain embodiments, I/O module 1208 is configured to provide graphicaldata to a display for presentation to a user. The graphical data may berepresentative of one or more graphical user interfaces and/or any othergraphical content as may serve a particular implementation.

In some examples, any of the facilities described herein may beimplemented by or within one or more components of computing device1200. For example, one or more applications 1212 residing within storagedevice 1206 may be configured to direct processor 1204 to perform one ormore processes or functions associated facilities 402 through 406 ofsystem 400. Likewise, storage facility 408 of system 400 may beimplemented by storage device 1206 or a component thereof.

In the preceding description, various exemplary embodiments have beendescribed with reference to the accompanying drawings. It will, however,be evident that various modifications and changes may be made thereto,and additional embodiments may be implemented, without departing fromthe scope of the invention as set forth in the claims that follow. Forexample, certain features of one embodiment described herein may becombined with or substituted for features of another embodimentdescribed herein. The description and drawings are accordingly to beregarded in an illustrative rather than a restrictive sense.

1. A system comprising: a processor; and a memory communicativelycoupled with the processor and storing instructions executable by theprocessor to: direct an imaging device to capture an image using aparticular exposure algorithm, wherein the image depicts an operationalscene illuminated by close-range light, and the particular exposurealgorithm is configured to minimize pixels that, when captured by animage sensor of the imaging device, are so bright that the image sensorsaturates or are so dim that the image sensor detects no appreciableamount of light, access a depth map of the operational scene, the depthmap including depth data corresponding to each pixel in the image,determine, based on the depth map, a far-range lighting coefficient foreach pixel in the image, the far-range lighting coefficient for eachrespective pixel determined based on the corresponding depth dataincluded in the depth map for that respective pixel, generate, based onthe image and the far-range lighting coefficient for each pixel in theimage, a processed image depicting the operational scene as beingilluminated by far-range lighting, and provide the processed image forpresentation on a display screen.
 2. The system of claim 1, wherein: theclose-range light illuminating the operational scene originates from aphysical light source associated with the imaging device and located ata first position a first distance from a particular point in theoperational scene; the far-range lighting illuminating the operationalscene as depicted in the processed image virtually originates from avirtual light source simulated to be located at a second position asecond distance from the particular point in the operational scene, thesecond distance greater than the first distance; and the instructionsare executable by the processor to determine, based on a ratio of thesecond distance to the first distance, the far-range lightingcoefficient for a particular pixel in the image that represents theparticular point in the operational scene.
 3. The system of claim 2,wherein the instructions are further executable by the processor to:receive, from a user of the system, user input representative of a userselection of the second position at which the virtual light source issimulated to be located; and define the second position based on theuser input.
 4. The system of claim 3, wherein: the instructions arefurther executable by the processor to provide, a slider input toolconfigured to facilitate the user selection of the second position; andthe instructions are executable by the processor to define the secondposition at which the virtual light source is simulated to be locatedbased on a setting of the slider input tool selected by the user.
 5. Thesystem of claim 1, wherein: the image captured by the imaging device isassociated with a first dynamic range; the instructions are executableby the processor to generate the processed image depicting theoperational scene as being illuminated by the far-range lighting bymultiplying a brightness value of each respective pixel in the imagedepicting the operational scene by the far-range lighting coefficientdetermined for that respective pixel; and the processed image depictingthe operational scene is associated with a second dynamic range that isless than the first dynamic range.
 6. The system of claim 1, wherein:the instructions are further executable by the processor to perform auniversal brightness adjustment on each pixel of the processed imagesubsequent to generating the processed image and prior to providing theprocessed image for presentation on the display screen; and theinstructions are executable by the processor to perform the universalbrightness adjustment by multiplying a brightness value of each pixel inthe processed image by a universal lighting coefficient.
 7. The systemof claim 1, wherein the instructions are executable by the processor toaccess the depth map of the operational scene by: accessing a firstimage and a second image from the imaging device, the first and secondimages depicting the operational scene from stereoscopic vantage pointshaving a predetermined spatial relationship; comparing the first imageto the second image; and generating the depth map based on thecomparison of the first image to the second image.
 8. The system ofclaim 7, wherein: the imaging device is a stereoscopic imaging devicethat includes a first image sensor and a second image sensor that isstereoscopic with the first image sensor such that the first and secondimage sensors are together configured to capture sets of synchronizedstereoscopic images; and the first and second images are included withina particular set of synchronized stereoscopic images captured by thefirst and second image sensors included within the stereoscopic imagingdevice.
 9. The system of claim 7, wherein: the first image is capturedat a first time by the imaging device; and the second image is capturedat a second time by the imaging device, the second time distinct fromthe first time.
 10. The system of claim 1, wherein: the image depictingthe operational scene is included within a sequence of images depictingthe operational scene, the sequence of images captured by the imagingdevice and further including a prior image that is captured within thesequence before the image; the instructions are executable by theprocessor to access the prior image before accessing the image; and theinstructions are executable by the processor to perform, based on theaccessed prior image, at least one of the accessing of the depth map andthe determination of the far-range lighting coefficient for each pixelin the image.
 11. (canceled)
 12. A system comprising: an imaging deviceconfigured to capture an image depicting an operational scene asilluminated by close-range light; a physical light source associatedwith the imaging device and configured to generate the close-range lightto illuminate the operational scene; a display screen configured topresent images captured by the imaging device; a processorcommunicatively coupled to the imaging device; and a memorycommunicatively coupled to the processor and storing instructionsexecutable by the processor to: direct the imaging device to capture,using a particular exposure algorithm, the image depicting theoperational scene as illuminated by the close-range light, wherein theparticular exposure algorithm is configured to minimize pixels that,when captured by an image sensor of the imaging device, are so brightthat the image sensor saturates or are so dim that the image sensordetects no appreciable amount of light, generate, based on the imagedepicting the operational scene, a depth map of the operational scene,the depth map including depth data corresponding to each pixel in theimage, determine, based on the depth map, a far-range lightingcoefficient for each pixel in the image, the far-range lightingcoefficient for each respective pixel determined based on thecorresponding depth data included in the depth map for that respectivepixel, generate, based on the image and the far-range lightingcoefficient for each pixel in the image, a processed image depicting theoperational scene as being illuminated by far-range lighting, andprovide the processed image for presentation on the display screen. 13.The system of claim 12, wherein: the physical light source is located ata first position a first distance from a particular point in theoperational scene; the far-range lighting illuminating the operationalscene as depicted in the processed image virtually originates from avirtual light source simulated to be located at a second position asecond distance from the particular point in the operational scene, thesecond distance greater than the first distance; and the instructionsare executable by the processor to determine, based on a ratio of thesecond distance to the first distance, the far-range lightingcoefficient for a particular pixel in the image that represents theparticular point in the operational scene.
 14. The system of claim 12,wherein: the image depicting the operational scene is associated with afirst dynamic range; the instructions are executable by the processor togenerate the processed image depicting the operational scene as beingilluminated by the far-range lighting by multiplying a brightness valueof each respective pixel in the image depicting the operational scene bythe far-range lighting coefficient determined for that respective pixel;and the processed image depicting the operational scene is associatedwith a second dynamic range that is less than the first dynamic range.15. The system of claim 12, wherein: the instructions are furtherexecutable by the processor to perform a universal brightness adjustmenton each pixel of the processed image subsequent to generating theprocessed image and prior to providing the processed image forpresentation on the display screen; and the instructions are executableby the processor to perform the universal brightness adjustment bymultiplying a brightness value of each pixel in the processed image by auniversal lighting coefficient.
 16. The system of claim 12, wherein: theimaging device is a stereoscopic imaging device that includes a firstimage sensor and a second image sensor that is stereoscopic with thefirst image sensor such that the first and second image sensors aretogether configured to capture sets of synchronized stereoscopic images;and the instructions are executable by the processor to generate thedepth map of the operational scene by: directing the stereoscopicimaging device to capture, by way of the first and second image sensors,a first image and a second image included within a particular set ofsynchronized stereoscopic images, the first and second images depictingthe operational scene from stereoscopic vantage points having apredetermined spatial relationship; comparing the first image to thesecond image; and generating the depth map based on the comparison ofthe first image to the second image.
 17. The system of claim 12,wherein: the image depicting the operational scene is included within asequence of images depicting the operational scene, the sequence ofimages captured by the imaging device and further including a priorimage that is captured within the sequence before the image; theinstructions are executable by the processor to access the prior imagebefore capturing the image by way of the imaging device; and theinstructions are executable by the processor to perform, based on theaccessed prior image, at least one of the generation of the depth mapand the determination of the far-range lighting coefficient for eachpixel in the image.
 18. (canceled)
 19. A method comprising: directing,by a lighting emulation system, an imaging device to capture an imageusing a particular exposure algorithm, wherein the image depicts anoperational scene illuminated by close-range light, and the particularexposure algorithm is configured to minimize pixels that, when capturedby an image sensor of the imaging device, are so bright that the imagesensor saturates or are so dim that the image sensor detects noappreciable amount of light; accessing, by the lighting emulationsystem, a depth map of the operational scene, the depth map includingdepth data corresponding to each pixel in the image; determining, by thelighting emulation system based on the depth map, a far-range lightingcoefficient for each pixel in the image, the determining of thefar-range lighting coefficient for each respective pixel performed basedon the corresponding depth data included in the depth map for thatrespective pixel; generating, by the lighting emulation system based onthe image and the far-range lighting coefficient for each pixel in theimage, a processed image depicting the operational scene as beingilluminated by far-range lighting; and providing, by the lightingemulation system, the processed image for presentation on a displayscreen.
 20. The method of claim 19, wherein: the close-range lightilluminating the operational scene originates from a physical lightsource associated with the imaging device and located at a firstposition a first distance from a particular point in the operationalscene; the far-range lighting illuminating the operational scene asdepicted in the processed image virtually originates from a virtuallight source simulated to be located at a second position a seconddistance from the particular point in the operational scene, the seconddistance greater than the first distance; and the determining of thefar-range lighting coefficient for a particular pixel in the image thatrepresents the particular point in the operational scene is based on aratio of the second distance to the first distance.
 21. The method ofclaim 19, further comprising performing, by the lighting emulationsystem, a universal brightness adjustment on each pixel of the processedimage subsequent to generating the processed image and prior toproviding the processed image for presentation on the display screen;wherein the performing of the universal brightness adjustment includesmultiplying a brightness value of each pixel in the processed image by auniversal lighting coefficient.
 22. The method of claim 19, wherein theaccessing of the depth map of the operational scene includes: accessinga first image and a second image from the imaging device, the first andsecond images depicting the operational scene from stereoscopic vantagepoints having a predetermined spatial relationship; comparing the firstimage to the second image; and generating the depth map based on thecomparison of the first image to the second image.